Method and apparatus for testing automated valuation models

ABSTRACT

A method and apparatus for real time testing of automated valuation using various indicators of accuracy. These indicators are then weighted according to their value as indicators of accuracy using individualized weighting factors or an equation. A ranking is then computed based upon the factors and their weights. This method is preformed continuously, so as to achieve real-time or periodically updated automated valuation model accuracy rankings.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/182,289, filed Jul. 13, 2011, titled “METHOD AND APPARATUS FORTESTING AUTOMATED VALUATION MODELS,” which is a division of U.S. patentapplication Ser. No. 11/007,750, filed Dec. 8, 2004, titled “METHOD ANDAPPARATUS FOR TESTING AUTOMATED VALUATION MODELS,” now U.S. Pat. No.8,001,024, each of which is hereby incorporated by reference herein inits entirety.

BACKGROUND

1. Field of the Invention

The present invention relates to testing procedures to evaluationproperty valuations, and more specifically to a method of and apparatusfor testing the accuracy of multiple automated valuation models.

2. Background of the Invention

Automated valuation models are increasingly being used in real estate asthe first or sole indicator of property value. The valuations providedby these automated valuation models are of varying accuracy. Currently,there exists no standardized means by which to compare the accuracy ofone automated valuation model to that of another. The “confidencescores” often included along with any automated valuation produced areonly indicators of a particular automated valuation model's internalevaluation that its own valuation is more or less accurate and oftenbear no relation to the internal evaluations of other automatedvaluation model valuations.

Therefore, a means by which these automated valuation models' accuracyat providing valuations would prove very useful. Providing a means togauge the accuracy of one automated valuation model against that ofanother enables the automated valuation consumer to choose thepreferable automated valuation model on which to rely. Generallyspeaking, automated valuation models may be more or less accuratedepending on the geographic area or price range of the property.Therefore, a means of testing the accuracy of automated valuation modelsthat is capable of categorizing valuations by geographic area, economictier, or property type would also be useful. There currently exists noempirically based ranking system for automated valuation model accuracy.Further, there currently exists no accurate and up-to-date data uponwhich to base such a ranking system.

It is therefore an object of this invention to provide a means by whichautomated valuation models may be compared against each other foraccuracy. It is a further object of this invention to provide this meansin particular sub-divisions such as geographic areas, price ranges,price tiers, and property types. It is a further object of thisinvention to provide a useful data set for comparing these automatedvaluation models and for evaluating the rankings computed using thismethod.

These and other objectives of the present invention will become apparentfrom the following description of the invention.

SUMMARY OF THE INVENTION

According to the present invention, a method and apparatus are describedwhereby an automated valuation model may be tested for accuracy. Thismethod and apparatus further provides a means by which comparisonsbetween various automated valuation models may be performed usingstandardized indications of accuracy. The method and apparatus of thisinvention also provides data from which any user of automated valuationmodel data may perform calculations and comparisons of whatever kindusing standardized data.

In the preferred embodiment of the invention, the method begins with thereceipt of new sales data on a property. However, data may be generatedin several ways in alternative embodiments including appraisals,foreclosures, and even other automated valuations. Once this new data isgenerated, it is quickly passed on to the real time testingimplementation to be described. This new data is inputted in real-time,as it is received, into a database. The data saved includes theproperty's sale (or other valuation) price and the property's geographicarea.

Immediately, the real time testing implementation requests an automatedvaluation of the new data's subject property. This request is forwardedto each automated valuation model to which the particular real timetesting implementation has access. Each automated valuation model thenreturns a valuation without reference to the current sale of thatsubject property. It is unlikely that an automated valuation model hasbeen updated to take this new property sale, appraisal, or othervaluation into account. This is due to the real time nature of theinvention. Because automated valuation models are not updated in realtime, it would be highly unlikely that any one automated valuation modelwould have already received and incorporated the new data that has justbeen received by the method of this invention.

Alternatively, an implementation of this invention may aggregateautomated valuation model valuation requests until the end of a day oruntil the end of any other predetermined period of time. Then, theimplementation of this invention requests a valuation for each propertyfor which new sale or reference data was generated since the lastrequest from each automated valuation model being ranked. The databaseof reference data and automated valuation model valuations is therebycontinuously updated, but need not necessarily be in real time. Thisautomated valuation model data will almost certainly not have beenupdated to include the new sales data because automated valuation modelsare usually only updated once a month or quarter. Whether valuationrequests are done in real time (immediately upon receipt of new saledata) or shortly thereafter (at the end of a predetermined time period),the continuously updated database of automated valuations and ofreference values provides a data set from which the data used to rankthe accuracy of automated valuation models can be derived.

The database containing the sale price (or other new data) and eachautomated valuation model valuation of the subject property may then beused to create various indicators of the accuracy of each automatedvaluation model's property valuations. These indicators may becalculated for any geographic area, property type, economic tier, or anyother characteristic of a property. In one embodiment of the invention,the data within the database is used to create several numericalindicators of the accuracy of an automated valuation model in a givengeographic area. The automated valuation models are then ranked, usingordinal numbers, according to their accuracy. These numerical indicatorsare then weighed, depending upon their usefulness as indicators ofoverall accuracy. Alternative weightings may be given to particularindicators. In another embodiment, the weighting of the variousindicators of accuracy may be done using an equation. Then, the rankingsmay be calculated based on price ranges or taking into account the costto request each automated valuation model valuation, such that a rankingmay be generated for the automated valuation model of the best accuracyfor the cost or the automated valuation model that is best in aparticular price range or geographic area. Additionally, secondaryrankings may be made as to accuracy excluding data referenced by theprimary (or most accurate) automated valuation model overall.Alternatively, the secondary rankings may be further subdivided toexclude only the data referenced by the primary (or most accurate)automated valuation model in a particular geographic area, economic tieror property type.

The rankings of accuracy in a particular geographic area, economic tieror property type may then be passed on to the users of automatedvaluations models. The rankings may be designed to cater to a particularclient, incorporating other interests, such as least-expensive accurateautomated valuation model. Additionally, the data itself indicating theaccuracy of each automated valuation model may simply be passeddirectly, either in total or piecemeal, to a user of automated valuationmodels. This data may be used to perform whatever calculations theparticular user desires in order to determine what automated valuationmodel is best for the particular user's needs. Alternatively, the datamay be passed along to an automated valuation model user along with theranking data as calculated in the preferred embodiment, such that theautomated valuation model user may ensure the accuracy of the rankingdatabase information and to perform their own calculations andweightings.

Further features and advantages of the present invention will beappreciated by reviewing the following drawings and detailed descriptionof the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a data structure used to implement the method and apparatus ofthe invention.

FIG. 2 is a more detailed depiction of specific portions of the datastructure depicted in FIG. 1 along with external components used in themethod of this invention.

FIG. 3 is a flowchart depicting the steps involved in the datacollection process for each property.

FIG. 4 is a flowchart depicting the steps involved in the periodicranking of automated valuation model accuracy and updating of databases.

FIG. 5 a is a table depicting an example of a primary ranking of severalautomated valuation models using a predetermined criterion.

FIG. 5 b is a table depicting an example of a secondary ranking of theremaining automated valuation models using the predetermined criterion.

FIG. 6 a is a table depicting a summary of the primary weighted rankingsof several automated valuation models using the data obtained from therankings show in FIG. 5 a.

FIG. 6 b is a table depicting a summary of the secondary weightedrankings of several automated valuation models using the data obtainedfrom the rankings shown in FIG. 5 b.

FIG. 7 a is a table depicting an example alternative embodiment of aprimary ranking of several automated valuation models using analternative predetermined criterion.

FIG. 7 b is a table depicting an example alternative embodiment of asecondary ranking of the remaining automated valuation models using thepredetermined criterion.

FIG. 8 a is a table depicting a summary of the primary weighted rankingsof several automated valuation models using the data obtained from therankings shown in FIG. 7 a.

FIG. 8 b is a table depicting a summary of the secondary weightedrankings of several automated valuation models using the data obtainedfrom the rankings shown in FIG. 7 b.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a method of and apparatus for real timetesting of automated valuation models (“AVMs” or singularly “AVM”).Referring first to FIG. 1, an example data structure for use inimplementing this method is depicted. Depicted in element 90 is anexample automated valuation model testing data structure (“datastructure”). The elements in this example data structure 90 may bealtered, and some may even be eliminated, without diminishing the scopeof the invention. This data structure 90 is one means by which toimplement the method of the invention. Many alternative embodiments ofthe present invention may be used which accomplish substantially thesame result.

The following components each may be implemented in software, inhardware, or a combination of both, so long as the method of thisinvention is carried out by the particular implementation. The controlprocessor 100 is used as an internal control mechanism. If implementedin software, for example, it is the main function of the program, usedin calling each subsidiary function. If implemented in hardware, it isthe chip in which each of the other functions is embodied. Theautomation processor 102 is used in automating the process by which thenew sale data or other new data is incorporated into the real timetesting database. It also automates the process by which automatedvaluation data is requested from each automated valuation model beingtested and the process by which periodic calculations and rankings ofthe various automated valuation models are performed. The categorizationprocessor 104 is used to categorize property according to its geographicarea, economic tier or property type. Economic tier refers to aparticular valuation range, price range, valuation tier or price tier ofwhich a property may be a part. Tiers may be divided into quartiles,made up of upper tier, two middle tiers and a lower tier. Tiers may becalculated as a percentile of valuation or price in relation to otherproperties in a large or small geographic area or a property type.Economic tier may be determined using any price or valuation relateddata concerning a particular property that would allow categorization ofa property with respect to another property.

Property type refers to a property type such as a single-familyresidence, condominium or any other means of categorizing properties.Alternatively, property type could also be a categorization based on thenumber of bedrooms and bathrooms in a particular property. Any method ofgrouping and differentiating properties based upon their individualcharacteristics could be considered a property type.

The calculation processor 106 is used to perform the calculationsassociated with deriving indicators of accuracy for each automatedvaluation model and it is also used in using those indicators and thevarious weighting factors to order the automated valuation models forranking. The comparison processor 107 is used in comparing the referencevalues with the automated valuations. Temporary data storage 108 is usedto temporarily store data as it is being used. In a standard personalcomputer, temporary data storage is random, access memory (RAM). It isused to store temporary data, such as partial calculations of theindicators of accuracy or ranking data before it is stored into the AVMaccuracy database 120. The input and output connectors 110 are used as abuffer between the data 90 and external data and output sources such asthe new data source 114.

The input and output connectors use standard communications protocolswhen able to do so and use proprietary communications protocols whennecessary to communicate with a particular data or output source. Theelements connected to the data structure 90 need not actually beseparate from the data structure 90. They are depicted as connected inthis example database, but some or all may be included within the datastructure 90 itself, thus eliminating the need for such a buffer forthat input or output source.

Data pertaining to a new property sale is provided as a new data source114. The new data source in the preferred embodiment is a continuallyupdated source of real estate new sales data. It need not necessarily beone source. The new data source 114 may be representative of manysources of sales data. In the preferred embodiment, the reference dataderived from the new data source 114 is an actual sale of a property. Inalternative embodiments, reference data from the new data source 114 maybe derived by using appraisals or from other automated valuation modelvaluations. The price data and valuation database 116 is also connectedto the data structure 90 through the input and output connectors 110.This is the database wherein all the new sale prices and each AVMvaluation for a subject property are stored. In the preferredembodiment, this price data is derived from new sales, but as statedabove, this data may also be from appraisals or from AVM valuations.This price data and valuation database 116 is depicted as a singledatabase. In alternative embodiments, it may be multiple databases, onefor each automated valuation model being tested and one for new saledata. Alternative categorization into multiple databases is possible inalternative embodiments of this invention.

The additional input and output resources 118 are also connected to thedata structure 90 of this invention using the input and outputconnectors. This element is representative of any additional input oroutput necessary or useful in the functioning of the real time testingdata structure 90. These additional input and output resources mayinclude a monitor, a keyboard, a mouse, a network connection, a modemconnection, a form of wireless connection, a direct connection toanother computer or network, an Internet connection for web access to aparticular database or user interface, or a connection to otherdatabases or data sources. These are only examples of potential inputand output resources. Many others are potentially possible and useful,though they are not depicted here.

The AVM accuracy database 120 is also connected to the data structure 90using the input and output connectors. This database need notnecessarily be separate from the data structure 90, but is depictedherein as being separate. This database stores the automated valuationmodel rankings, which in the preferred embodiment are updated in realtime, of each automated valuation model being tested. This database isrepresented as being a single entity. It may also be divided andsubdivided into accuracy databases for particular geographic areas,economic tiers, and property types. Individual AVM accuracy databases120 may be created for any identifiable grouping of properties for whichdata is available.

The automated valuation model connector 112 is used to connect the datastructure 90 implementation of this invention with the multipleautomated valuation models being tested. It performs a function similarto that of the input and output connectors, specifically designed withmultiple automated valuation model valuation requests and responses inmind. In the preferred embodiment, the transactions between an automatedvaluation model and the invention, such as AVM X valuation request 128and AVM X valuation response 130, are performed using extended markuplanguage (XML). Often used in communications between different softwareprograms or data structures, this language is used by the automatedvaluation model connector 112 to format the valuation request for eachautomated valuation model being tested. This XML request is then sent tothe particular automated valuation model being tested, the AVM thenprovides its response 130 in XML to the automated valuation modelconnector 112 which then translates it back into a form readable by therest of the automated valuation model testing data structure 90. Exampleautomated valuation models AVM X 122, AVM Y 124, and AVM Z 126 aredepicted as connected to the automated valuation model connector. Eachof these AVMs may have a different format in which it expects automatedvaluation requests and responses to be forwarded and returned. Theautomated valuation model connector 112 is designed to handle and formatrequests and responses for each of these AVMs. These three connectedAVMs are only examples. Additional AVMs may be connected. In thepreferred embodiment, all available AVMs are connected to enable thereal time testing method to provide the most complete data and accuracyrankings for AVMs.

Referring now to FIG. 2, specific portions of the data structure alongwith external elements are depicted. This depiction contains several ofthe elements depicted in FIG. 1, but includes the more direct ways inwhich the components of this method are related. The first step in themethod of this invention is to obtain new data or a reference valuegenerated from a new data source 114. As stated above, this is generallya new sale of a property, but may in alternative embodiments be anappraisal or even a particular automated valuation model valuation (foruse in comparing other AVMs to the one used as the new sale data). Thenew data is inputted through the input and output connectors 110 intothe new data and valuation database 116. The second step involvesrequesting and receiving automated valuations from each of the automatedvaluation models being tested for each reference value. This request mayoccur immediately subsequent to the reception of new automated valuationmodel data or at some time shortly thereafter. In the preferredembodiment, this occurs within one day to one week from the receipt ofnew sale or reference data. In this figure, example AVMs X 122, Y 124and Z 126 are depicted. Once the new data and the AVM valuations arereceived through the automated valuation model connector 112, they aresent through the input and output connectors 110 and stored in the newdata and valuation database 116. The aggregate data is then usedperiodically or on command by a user by the calculation processor tocalculate various indicators of accuracy for each AVM and to rank eachAVM by the various indicators, to weight them according to weightingfactors or a weighting equation, and then rank the automated valuationmodels accordingly.

The various indicators of accuracy are calculated as percentages. Theseindicators are: absolute mean error, absolute median variance to thereference value (usually the new sale price), mean squared error,automated valuation model accuracy (defined as the percentage ofvaluations within a predetermined percentage of the reference value,such as 10% of the reference value), the percentage incidence of outliervalues (defined as valuations that are beyond a predetermined percentageof the reference value, such as 25% more or less than the referencevalue), and hit rate percentage (defined as percentage of properties forwhich the AVM was able to return an evaluation).

These indicators of accuracy are calculated periodically or on commandby a user based upon the aggregate data collected over time using theabove process of requesting a valuation for a property for which newsale data or a reference value has been generated. Over time, this willprovide an increasingly more accurate picture of a particular automatedvaluation model's accuracy overall and in a particular geographic area,economic tier, or property type. Additionally, these indicators ofaccuracy may be calculated for particular time frames, for example, ifsome improvements have been made recently to a particular automatedvaluation model, then the indicators of accuracy and subsequent rankingsdependant upon them may be calculated for the time since theimprovements were made.

In the embodiment using weighting factors for each of the indicators ofaccuracy, the weighting associated with each of these factors is alsoimportant. In this embodiment, the weighting factors are as follows:

hit rate is multiplied by 0.05 or 5%

median error is multiplied by 0.1 or 10%

absolute mean error is multiplied by 0.1 or 10%

mean error squared is multiplied by 0.1 or 10%

automated valuation model accuracy is multiplied by 0.35 or 35%

outlier percentage is multiplied by 0.30 or 30%

Hit Rate Percentage

Hit rate percentage (h) is calculated using the following equation:h=n″/n

Where n″ is the number of properties for which a valuation was returnedand n is the total number of properties.

Median Error

The individual property errors, used in calculating the median error tothe reference values are calculated using the equation:

error=|s _(n) −r _(n) |/s _(n) for every sale price (s) and referencevalue (r).

Where

s_(n) is the sale price or new data price for property n,

r_(n) is the reference value price for property n, and

n is the number of properties used in the calculation.

These errors are then ordered in ascending order such that the lowest isi₁ and the highest is i_(n).

Then, the median error to the reference value (m) is calculated usingthe equation:

m=i _(n/2)

Where n is the number of properties used and i is the list of thoseerrors listed in ascending order by absolute value.

Absolute Mean Error

The absolute mean error (a) is calculated using the equation:

a=Σ(|s _(n) −r _(n) |/s _(n))/n

Where

s_(n) is the sale price or new data price for property n,

r_(n) is the reference value price for property n, and

n is the number of properties used in the calculation.

The summation in the previous equation is calculated from 1 to n, where1 is the first subject property and n is the last.

Mean Squared Error

The mean squared error (s) is calculated using the equation:

s=Σ[(s _(n) −r _(n))² /s _(n) ]/n

Where s_(n) is the sale price or new data price for property n, wherer_(n) is the reference value price for property n, and where n is thenumber of properties used in the calculation.

Automated Valuation Model Accuracy

Automated valuation model accuracy (c) is calculated using the equation:

c=n′/n

Where n′ is the number of properties whose valuation was within apredetermined percentage of the reference value, such as 10% of thereference value, and where n is the total number of properties. Thispercentage may be greater or smaller without affecting the method ofthis invention.

Percentage Incidence of Outlier Values

Percentage incidence of outlier values (o) is calculated using thefollowing equation:

o=o′/n

Where o′ is the number of properties whose valuation was 25% more orless than the reference value and n is the total number of properties.The percentage that is considered to be an outlier value may be greateror smaller without affecting the method of this invention.

Each of the above calculations results in a decimal number that isrepresentative of a percentage value. Depending on the indicator ofaccuracy a large percentage may indicate accuracy or may indicateinaccuracy. A large percentage, for example in hit rate percentage,indicates accuracy. In one embodiment, the highest hit rate percentageis ranked number one in a particular geographic area, economic tier, orproperty type. In other indicators of accuracy, for example absolutemean error, a lower percentage indicates more accuracy, stating that theaverage percentage of value that this particular automated valuationmodel “misses” the target. For this indicator and in this embodiment,the lowest percentage is ranked highest. In each of the categories, eachautomated valuation model is ordinally ranked as compared with everyother automated valuation model. In this embodiment, these rankings arethen multiplied, for each of the automated valuation models in eachgeographic area, economic tier, or property type, by the associatedweighting factor. Each of these multiplied values are added together andmultiplied by a scaling factor, such as ten, which results in an“accuracy score.” The lowest accuracy score in a particular geographicarea, economic tier or property type is the most accurate automatedvaluation model.

Additionally, once the automated valuation models have been ranked,according the above process, a secondary ranking may take place. Duringthis ranking, the same reference values, usually sale prices, butsometimes alternative new price data, are considered. The computation ofrankings is performed using data for which the primary ranked AVM didnot return an evaluation. For example, the hit rate is calculated forthe primary ranking using all reference values and automated valuationmodel valuations for each subject property. The secondary hit rate iscalculated using only reference values and automated valuation modelvaluations for subject properties for which the primary automatedvaluation model did not return a valuation. Similarly, the absolute meanerror would only be calculated for properties for which there was novaluation returned by the primary ranked automated valuation model inthat geographic area, economic tier, or property type. The purpose ofthis secondary ranking is to help enable automated valuation model usersto determine which automated valuation model to use second, if theprimary ranked automated valuation model is unable to value a particularsubject property.

In the next step, any external client databases 136 are updated. Thisconnection is made through the input and output connectors 110. Theexternal client database 136 is a database of a client to which the newrankings and accuracy indicators and weights are sent. Additionally, thenew data generated by the calculation processor 106 and the comparisonprocessor 107 is sent through the input and output connectors 110 andappended to the AVM accuracy database 120. Alternatively, the rankings,reference values and automated valuations may be added to a new AVMaccuracy database 120 for only a particular time period, such as athree-month period. In either database update, the database 120 isupdated to reflect changes in the rankings of automated valuation modelsand the numerical changes to the various indicators of accuracy.Additional data users 138 may also be updated through the input andoutput connectors 110. These may include the users of the raw data, suchas the indicators of accuracy or simply raw data from the new data andvaluation database 116 to be used to perform their own calculations.Finally, the new data source 114 will likely be used by relevant publicagencies to update public record databases 132. Additionally, variousautomated valuation model venders periodically use this new data,generally on a monthly or quarterly basis, to update the databases ofeach of the automated valuation models 134.

FIG. 3, depicts the steps involved in the data collection process of thepresent invention. The first step in this process is to gather orgenerate new property value indication data 140. This entails thereception of new sale data from publicly available records. In thepreferred method of this invention, the data is appended to the databaseas quickly as it is received. Next, this new data is received and stored142 in the new data and valuation database 116. Next, valuation requestsare made to each automated valuation model to be tested 144. Thisrequest need not occur immediately after receipt of new sale data. Inthe preferred embodiment of the invention, the new sale data isaggregated and all automated valuations are requested at the end of aday. In alternative embodiments, the requests 144 are made immediatelyupon receipt of new sale data or at any other predetermined timeinterval such as bi-daily or weekly. Using either embodiment, thedatabase which stores the automated valuation model valuations and theactual sale price or other reference data is continuously updated andbecomes increasingly useful for deriving measures of the accuracy ofeach automated valuation model.

In the preferred embodiment, the valuation data provided by eachautomated valuation model is done without reference to the very recentsale. New sale data is gathered using this method for both the purposeof testing automated valuation models but also to provide that new saledata to automated valuation models. Most automated valuation modeldatabases are only updated monthly or quarterly with the data providedusing the aggregate new sale data gathered using this method. Therefore,each automated valuation model will most likely provide an automatedvaluation for the new sale data subject property without reference tothe most recent sale price because it has not yet been provided. Even ifa particular automated valuation model is updated on a weekly or even adaily basis, the present invention preferably provides accuracy testingof the automated valuation models prior to providing data of the mostrecent property sales for updates to automated valuation models. Thisenables the method of this invention to most effectively gauge theaccuracy of each automated valuation model being tested. Once eachrequest is made and each automated valuation model responds with itsvaluation, this data is received and stored 146 into the same databaseas the new data. Alternatively, this data may be stored in a separatedatabase from that of the new data or in individual databases for eachautomated valuation model being tested.

Referring now to FIG. 4, a flowchart depicting the steps involved in theperiodic ranking and AVM accuracy database update are depicted. Thefirst step is to calculate the accuracy indicators 148. This takes placein the preferred embodiment in the calculation processor 106. Each ofthe indicators of accuracy are calculated for each automated valuationmodel. This is done using the aggregate reference values and automatedvaluations for each subject property over a set period of time. Thistime period may be over the life of a particular automated valuationmodel or for using all data that the method of this invention has beenable to store. Alternatively, these indicators of accuracy may becalculated for only a smaller subset of all of the data availablebecause of the method of this invention. Each of the indicators ofaccuracy is a percentage or an average percentage of a plurality ofvaluations that may be used to show, in general, the accuracy of each ofthe tested automated valuation models.

Next, in one embodiment, each AVM is ranked in each indicator ofaccuracy and in each category-geographic area, economic tier, orproperty type-by ordinal numbers, then the ordinal number for each AVM'sranking is multiplied by its corresponding weighting factor, and theproducts of these multiplications are summed 150. The corresponding sumof the products is then used to rank the automated valuation models 152in the embodiments described. In the embodiment that uses ordinalrankings multiplied by weighting factors, the lowest summed number isthe most accurate automated valuation model, and the next lowest summednumber is the next most accurate.

In the alternative embodiment that uses a weighting equation, an overallaccuracy indicating score is created using an additional formula, ratherthan ranking each automated valuation model compared to another. In thisembodiment, each of the accuracy indicators is calculated in much thesame way, but they are weighted using a formula or equation. In thealternative embodiment, the highest calculated number is the mostaccurate automated valuation model, and the next highest calculatednumber is the next most accurate. The automated valuation models arethen ranked according to which has the highest accuracy score. Thisalternative embodiment will be described further in discussing FIGS. 7a-8 b.

Finally, in the embodiments described, once the rankings are calculated,the method of this invention updates the AVM accuracy database 154. Thisdatabase then contains the most up to-date rankings and the indicatorsthat led to those rankings. Referring now to FIG. 5 a, the contents of aprimary ranking database are depicted. The various indicators ofaccuracy are depicted in column 155. These are hit rate percentage 156,median error 158, absolute mean error 160, mean squared error 162, AVMaccuracy 164, and outlier percentage 166. Hit rate percentage 156 refersto the percentage of reference properties for which each automatedvaluation model was able to return a valuation. Median error 158 refersto the percentage error at the middle range of the ordered “list” oferrors with reference to the reference value. Absolute mean error 160refers to the percentage average of the absolute value of the list oferrors in valuation. Mean squared error 162 is the percentage average inrelation to the reference value of the squares of the “list” of errorsin valuation. AVM accuracy 164 refers to the percentage of automatedvaluations within plus or minus 10% of the reference value. Outlierpercentage 166 refers to the percentage of automated valuations that aremore than plus or minus 25% of the reference value.

Also depicted in element 168 in the embodiment shown in FIG. 5 a is thescore or accuracy score of the particular automated valuation model inthe particular geographic area. The rank for each evaluated AVM based onfactors 156 through 166 for a predetermined geographic area is in column170. The rankings for three AVMs, X, Y, and Z, are shown in FIG. 5 a forthree states, Florida, California, and New Jersey. A lower rankingnumber indicates higher accuracy. For example, the rankings for threeAVMs for Florida are shown, AVM X 172, AVM Y 174, and AVM Z 176. Thevarious factors are compared and then ranked. For example in element178, the hit rate percentage of AVM X in Florida is 93.6%. For AVM Y inFlorida, depicted in element 180, the hit rate percentage is 85.1%. ForAVM Z in Florida, the percentage is 67.8%. Because the higher the hitrate, that is the number of properties for which a response wasgenerated by the automated valuation model, the higher the accuracy, alarger percentage for this factor is given a higher rank. Therefore, AVMX is ranked #1 in element 184, AVM Y is ranked #2 in element 186 and AVMZ is ranked #3 in element 188.

Similarly, AVM Accuracy, depicted in element 164, is returned as apercentage. This represents, as stated above, the percentage of valuesthat were within plus or minus 10% of the actual price or otherreference value. The percentage for AVM X of accuracy was 49.2%,depicted in element 190. The same percentage for AVM Y was 53.4%,depicted in element 192. The same percentage for AVM Z was 50.5%,depicted in element 194. This results in a ranking of #3 for AVM X inelement 196, of #1 for AVM Y in element 198 and a ranking of #2 for AVMZ in element 200.

These rankings are then aggregated using a particular weighting appliedto each factor. In alternative embodiments, as stated above, differentfactors may be used. Additionally, alternative weightings may be used inorder to improve the usefulness of the rankings or to provide differenttypes of results. Specific alternative embodiments may be implemented toprovide a particular type of result. In the first embodiment, theweighting factors are as follows:

hit rate is multiplied by 0.05 or 5%

median error is multiplied by 0.1 or 10%

absolute mean error is multiplied by 0.1 or 10%

mean error squared is multiplied by 0.1 or 10%

automated valuation model accuracy is multiplied by 0.35 or 35%

outlier percentage is multiplied by 0.30 or 30%

The results of these multiplications are then added together and thatsum is multiplied by a scaling factor such as ten to arrive at anaccuracy score or score as depicted in element 168.

For example, the scaled result for AVM X in Florida, with the weightingfactors multiplied by their corresponding rankings, is computed asfollows:

Score=10*((0.05*1)+(0.10*3)+(0.10*3)+(0.10 3)+(0.35*3)+(0.30*1))

which equals

10*(0.05+0.30+0.30+0.30+1.05+0.30)

which equals

23 which is the reported score in element 202.

Using the same computation procedure, the score for AVM Y in Florida is13.5, as depicted in element 204. Similarly, the score for AVM Z inFlorida is 23.5, as depicted in element 206.

The same factors, weightings and scores are calculated for eachgeographic area, economic tier, or property type. Other states, such asCalifornia or New Jersey are depicted in elements 208 and 210respectively. The rankings and calculated percentages for both statesare also depicted in the remainder of this table. The contents of thistable may be all or part of the content delivered to third partyrequesters. However, the database of properties valuations and referencevalues is maintained, along with the data depicted here in thisembodiment. The embodiments described update the database quarterly ormonthly. Many other automated valuation models may have data stored andranked accordingly using the method of this invention.

Referring now to FIG. 5 b, the contents of an exemplary secondaryranking database are depicted. A secondary ranking provides a ranking ofthe automated valuation models for properties for which the automatedvaluation model ranked first in the primary ranking was unable to returna valuation. This ranking is therefore useful in determining whichautomated valuation model is a good “second choice.” This databaserepresents most of the same calculations as those of the primary raking,minus the data that was already calculated using the primary rankedautomated valuation model, that is the automated valuation model thatranked number 1 overall in a particular category or geographic area.Similar factors are considered along the left column. These are hit ratepercentage 212, median error 214, absolute mean error 216, mean squarederror 218, AVM accuracy 220 and outlier percentage 222. This secondaryranking is used to determine which automated valuation model to useafter the primary ranked model. There is an importance in providing anyvaluation after the primary ranked automated valuation model was unableto provide one. The hit rate percentage of 70.3% is depicted, as shownin element 213. Also depicted is the score 224. Along the top, the rank226 or 230, the automated valuation model being tested, such as AVMX—Florida 228 or AVM Z—Florida 232 are depicted. In the firstembodiment, as indicated above, the weighting factors are as follows:

hit rate is multiplied by 0.05 or 5%

median error is multiplied by 0.1 or 10%

absolute mean error is multiplied by 0.1 or 10%

mean error squared is multiplied by 0.1 or 10%.

automated valuation model accuracy is multiplied by 0.35 or 35%

outlier percentage is multiplied by 0.30 or 30%

The results of these multiplications are then added together and thatsum is multiplied by a scaling factor such as ten to arrive at anaccuracy score or score as depicted in element 224.

For example, the scaled result for AVM X in Florida, with the weightingfactors multiplied by their corresponding rankings, is computed asfollows:

Score=10*((0.05*1)+(0.10*1)+(0.10.*1)+(0.10*1)+(0.35 1)+(0.30*1))

which equals

10*(0.05+0.10+0.10+0.10+0.35+0.30)

which equals

10 which is the reported score in element 234.

Using the same computation procedure, the secondary ranking score forAVM Z in Florida is 20, as depicted in element 236.

Referring now to FIG. 6 a, a summary of the example primary weightedranking is depicted. This is a summary form of the results tabulated inFIG. 5 a. In the left column, each automated valuation model 238 isdepicted. The Florida ranking 240 is shown at the top. The score 242,from FIG. 5 a and rankings, based upon the score from FIG. 5 a aredepicted. The automated valuation model with the lowest score in aparticular area is the most accurate in that area, according to oneembodiment of this invention. For example, AVM Y 246 with a score of13.5, depicted in element 248 is ranked #1 in element 250. AVM X 252with a score of 23, depicted in element 254 is ranked #2 in element 256.AVM Z 258 with a score of 23.5, depicted in element 260 is ranked #3 inelement 262. This consolidated table can be used to describe therelative accuracies of each automated valuation model in relation toeach other in a particular geographic area (in this case) or other suchdivision of properties. Many other automated valuation models may beincluded in this table and ranked accordingly.

Referring now to FIG. 6 b, an example summary secondary weighted rankingis depicted. This table is similar to the table depicted in FIG. 6 a,but is based on the data contained in FIG. 5 b. Therefore, this dataexcludes the valuations that were theoretically done using the primaryranked automated valuation model that is the most accurate one in aparticular geographic area. This weighted ranking displays in thetabular form, automated valuation model 264, score 266, and rank 268.The Florida ranking is depicted in element 270. In Florida, AVM X 272had a secondary score of 10, depicted in element 274 and a ranking of#1, depicted in element 276. All of the remaining automated valuationmodels are ranked in this secondary ranking.

Referring now to FIG. 7 a, the results of an alternative embodiment ofthe method used to implement the primary ranking of this invention isdepicted. In this embodiment of the present invention, a formula is usedto provide the relative weightings of the individual indicators ofaccuracy. The elements depicted in FIG. 7 a are much the same as thosedepicted in FIG. 5 a. The indicators of accuracy hit rate percentage 278through outlier percentage 280 are the same. The values for hit ratepercentage 278 in each automated valuation model are the same as in FIG.5 a. For example, the hit rate percentage for AVM X in Florida is 93.6%as depicted in element 288. Similarly, the outlier percentage, depictedin element 290 is 12.7%. Each of the same automated valuation models aredepicted; AVM X in 282, AVM Y in 284 and AVM Z in 286. The formulas usedto calculate each of the percentage values are the same. However, anoverall accuracy indicating score 292 is created using an additionalformula, rather than ranking each automated valuation model's accuracyindicators compared to another. The following is the score computationequation where each percentage is represented as a decimal:

Score=f*((h+c)−(m+a+s+o))

Where

f=multiplier factor, in the preferred embodiment it is 500

h=hit rate percentage

c=automated valuation model accuracy

m=median error to the reference value

a=absolute mean error s=mean squared error

o=percentage incidence of outlier values

Using this formula, along with the data in element 288 through element290, the score in element 294 is calculated. The equation, with thenumbers in place of the variables would appear as follows:

Score=500*((0.936+0.492)−(0.073+0.054+0.003+0.127))

which equals

500*(1.428−0.257)

which equals

585 which is the reported score in element 294.

This equation is used to calculate each of the scores for the automatedvaluation models in each geographic area, economic tier or propertytype. A multiplier factor of 500 is used in the invention to accentuatethe differences in accuracy between the automated valuation modelvaluations. A higher score represents a more accurate automatedvaluation in a particular geographic area, economic tier or propertytype. In Florida, for example AVM X is the most accurate with a score of585, depicted in element 294. AVM Y is the next-most accurate with ascore of 566, depicted in element 296 and AVM Z is the least accuratewith a score of 455, depicted in element 298.

Referring now, to FIG. 7 b, the results of an alternative embodiment ofthe method used to implement the secondary ranking of this invention isdepicted. As stated above, the secondary ranking is useful in finding a“second choice” automated valuation model. The valuations performed bythe first-ranked automated valuation model in the primary ranking areremoved. In this example, the “overall” primary automated valuationmodel was removed. This means that because AVM Y was the most accurateoverall, it was “chosen” as the primary automated valuation model foreach region. In an alternative method, the primary automated valuationmodel may be chosen for a particular geographic region, economic tier orproperty type. In that embodiment, for example, in Florida, AVM X wouldhave been the primary automated valuation model because it was moreaccurate in that geographic area. So, it would be excluded from thesecondary ranking.

In this embodiment, only the valuations for which the top ranked primaryautomated valuation model, AVM Y, did not return a value are considered.In this table many of the same elements depicted in FIG. 5 b are shown.Indicators of accuracy from hits 300 to outlier percentage 302 aredepicted. Both of the previous non-top-ranked automated valuation modelsare included, AVM X 304 and AVM Z 306.

As stated above, hit rate percentage is again depicted because for asecondary ranking, the percentage of additional properties for whichvaluations are returned is important in deciding which automatedvaluation model to use as a “second choice.” In this table, the hit ratepercentage for AVM X in Florida was 70.3%, depicted in element 308. Inan alternative embodiment of this implementation, the total number ofthe remaining properties that were able to be valued may be considered.

The same equation used for the primary ranking is used again, excludingthe previously valued properties. The equation is as follows:

Score=f*((h+c)−(m+a+s+o))

Where

f=multiplier factor, in the preferred embodiment it is 500

h=hit rate percentage

c=automated valuation model accuracy

m=median error to the reference value

a=absolute mean error

s=mean squared error

o=percentage incidence of outlier values

The equation for AVM X in Florida with the numbers in place of thevariables would appear as follows:

Score=500*((0.703+0.571)−(0.098+0.077+0.006+0.238))

which equals

500*(1.274−0.419)

which equals

427 which is the reported score in element 316.

As stated above, a higher score denotes a higher level of accuracy. Thescore of AVM Z in Florida, depicted in element 318 is only −373 and itis therefore considerably less accurate than AVM X in Florida in theproperties remaining after the top ranked automated valuation model inthe primary ranking has valued properties. Both the primary andsecondary ranking of automated valuation model accuracy using either ofthe embodiments provides a means by which automated valuation models maybe compared against each other for accuracy that are improvements overthe prior art.

Referring now to FIG. 8 a, an example summary primary weighted rankingof the alternative embodiment is depicted. This is a summary form of thedata presented in FIG. 7 a depicting only the scores in each locationand the relative ranking. Similar to FIG. 6 a above, the state for whichthe rankings are being done is depicted as Florida in element 320. Thecolumn for automated valuation model 322, includes AVM X 328, AVM Y 330and AVM Z 332. The respective scores for each of these automatedvaluation models are 585, depicted in element 334; 566, depicted inelement 336; and 455, depicted in element 338. Therefore, AVM X 328 withits score of 585 in Florida is ranked number one, as depicted in element340. The overall rankings depicted below are a result of the addition ofeach of the scores in each area. AVM Y 342 with its overall score of1861, depicted in element 344 is ranked number one 346 overall. In analternative embodiment, the overall ranking may be determined bydividing the geographic area score by the number of properties in thatarea for which automated valuations were provided and then adding thenumbers together. This would provide a ranking weighted as to thepercentage of homes in a particular market or its relative importance inuse of automated valuation data.

Referring now to FIG. 8 b, a secondary weighted ranking is depicted.This is a summary table of the data contained in FIG. 7 b. This table isuseful in comparing directly secondary rankings. As in FIG. 6 b, theautomated valuation model being ranked is depicted in the column denotedby element 350. The Florida secondary ranking, for example, is depictedin element 348. AVM X in Florida, for example, depicted in element 354,received a rank of one, as depicted in element 356. The column depictedin element 352 denotes the rank. Overall, AVM X 358, received the numberone ranking 360 with a score of 1079, depicted in element 362.

A method of and apparatus for real time testing of automated valuationmodels has been described. It is to be understood that the foregoingdescription has been made with respect to specific embodiments thereoffor illustrative purposes only. The overall spirit and scope of thepresent invention is limited only by the following claims, as defined inthe foregoing description.

1. (canceled)
 2. A method comprising: providing ranking criteria to anautomated valuation model testing entity; accessing, by a computersystem comprising computer hardware, rankings data and accuracy dataassociated with a plurality of automated valuation models, and providedby the automated valuation model testing entity, wherein the rankingsdata comprises rankings of the plurality of automated valuation modelsbased at least in part on the provided ranking criteria and comparisonsof valuations for a subject property received from the plurality ofautomated valuation models without reference to new valuation dataassociated with the subject property; and performing, by the computersystem, one or more calculations based at least in part on the accessedrankings data.
 3. The method of claim 2, wherein the ranking criteriacomprises cost preferences associated with automated valuation models.4. The method of claim 2, wherein said accuracy data comprises dataindicative of accuracy associated with each automated valuation model.5. The method of claim 2, wherein the accuracy data comprises weightingsassociated with the accuracy data.
 6. The method of claim 2, wherein theautomated valuation model testing entity provides the rankings data inreal-time.
 7. The method of claim 2, wherein the automated valuationmodel testing entity provides the rankings data periodically.
 8. Themethod of claim 2, wherein the new valuation data comprises propertysale data.
 9. The method of claim 2, wherein the new valuation datacomprises an appraisal for the subject property.
 10. The method of claim2, wherein the new valuation data comprises an automated valuation ofthe subject property.
 11. A system comprising: physical data storageconfigured to store (1) a reference valuation for a subject property and(2) a plurality of automated valuations of the subject property, eachautomated valuation generated by an automated valuation model (AVM)without use of the reference valuation for the subject property, whereindifferent automated valuations correspond to different AVMS; and acomputer system in communication with the physical data storage, thecomputer system comprising computer hardware, the computer systemprogramed to: provide ranking criteria to an automated valuation modeltesting entity; access rankings data and accuracy data associated with aportion of the plurality of automated valuation models, and provided bythe automated valuation model testing entity, wherein the rankings datacomprises rankings of the portion of the plurality of automatedvaluation models based at least in part on the provided ranking criteriaand comparisons of valuations for the subject property received from theportion of the plurality of automated valuation models; and perform oneor more calculations based at least in part on the accessed rankingsdata.
 12. The system of claim 11, wherein the ranking criteria comprisescost preferences associated with automated valuation models.
 13. Thesystem of claim 11, wherein the automated valuation model testing entityprovides the rankings data in real-time.
 14. The system of claim 11,wherein the automated valuation model testing entity provides therankings data periodically.
 15. The system of claim 11, wherein thereference valuation comprises property sale data.
 16. The system ofclaim 11, wherein the reference valuation comprises an appraisal for thesubject property.
 17. The system of claim 11, wherein the referencevaluation comprises an automated valuation of the subject property. 18.A tangible computer-readable medium that stores thereon a plurality ofcomputer-executable instructions configured, when executed by a computerprocessor, to cause computer hardware to perform operations comprising:accessing rankings data and accuracy data associated with a plurality ofautomated valuation models, and provided by the automated valuationmodel testing entity, wherein the rankings data comprises rankings ofthe plurality of automated valuation models based at least in part onthe provided ranking criteria and comparisons of valuations for asubject property received from the plurality of automated valuationmodels without reference to new valuation data associated with thesubject property; testing the rankings data to determine accuracy of therankings data; and performing one or more calculations based at least inpart on the accessed rankings data.
 19. The tangible computer-readablemedium of claim 18, wherein the automated valuation model testing entityprovides the rankings data in real-time.
 20. The tangiblecomputer-readable medium of claim 18, wherein the new valuation datacomprises property sale data.
 21. The tangible computer-readable mediumof claim 18, wherein the reference valuation comprises an appraisal forthe subject property.